Automated designation of tie-points for image-to-image coregistration

Image-to-image registration requires identification of common points in both images (irnage tie-points: ITPs). Here. we describe software implementing an automated, area-based technique for identifying ITPs. The ITP software was designed to follow two strategies: ( I ) capitalize on human knowledge and patternrecognition strengths, and (2) favour robustness in many situations over optimal performance in any one siluation. We tested the software under several confounding conditions, representing image distortions, inaccurate user input, and changes between images. The software was robust to additive noise, moderate change between images, low levels of image geometric distortion, undocumented rotation, and inaccurate pixel size designation. At higher levels of image geometric distortion, the software was less robust, but such conditions are often visible to the eye. Methods are discussed for adjusting parameters in the ITP software to reduce error under such conditions. For more than 1600 tests, median time needed to identify each ITP was approximately 8s on a common image-processing computer system. Although correlation-based techniques--such as those implemented in the free software documented here--have been criticized, we sugges! that they may. in fact, be quite useful in many situations for users in the remote sensing community. !. Introduction When two images of the same area arc to be analysed, they must be geometrically registered to one another. Image-to-image registration involves: (1) the identification of many image tie-points (ITPs), followed by (2) a calculation based on those points for transforming one ilnage's pixel coordinates into the coordinate space of the other image, followed finally by (3) a resampling based on that transformation. While steps 2 and 3 are relatively standardized processes, the manual identification of image tie-points may be time-consuming and expensive, and includes bias of the interpreter. For these reasons, automated techniques for detection of ITPs have been developed that are potentially cheaper and repeatable. : The variety of automated registration techniques has been amply summarized elsewhere (Bernstein 1983, Brown 1992, Fonseca and Manjunath 1996, Schowengerdt 19971. Briefly, the techniques to identify tie-points in two images fall into two main huernational Journal of Remole Sensing ISSN 0143-1161 print/ISSN 1366-5901 online 4;) 2003 Taylor & Francis Ltd htt p://www.tand f.co.u k/journals DO I:, I 0.1080/0143116021000024249 3468 R. E. Kemled)~ aml W. B. Cohen categories: area-based matching and feature-based matching. Area-based matching approaches compare spatial patterns of pixel grey-scale values (digital numbers) in small image subsets (Pratt 1974, Davis and Kenue 1978). Match-points between image subsets are located by maximizing some function of similarity between an image subset in one image and the image subset in another image. The ftmction of similarity may be a modified difference function or a more complex correlation function (Bernstein 1983, Schowengerdt 1997). Variations on this basic theme exist that seek to decrease computation time (Barnea and Silverman 1972) or to compute a simultaneous solution for multiple points, rather than using sequential image subsets (Rosenhohn 1987, Li 1991). in contrast to area-based matching, featurebased matching does not attempt to directly match grey-scale pixel patterns between the two images. Rather, pattern-recognition algorithms are applied to grey-scale pixel patterns within each image separately to identify prolninent spatial features, such as edges (Davis and Kenue 1978. Li et al. 1995) or unique shapes (Tscng et al. 1997). These features are described in terms of shape or pattern parameters that are independent of the image grey-scale values (Goshtasby et aL 1986, Tseng et aL 1997). The separate features in the two images are then matched based on those shape and pattern parameters. Although these techniques have been in existence for decades and are welldocumented in image-processing tcxts (Pratt 1991, Schowengerdt 1997), many users of satellite data still rely on manual designation of tie-points. To assess use of automated techniques, we surveyed every article published in the International Journal of Remote Sensing for the year 2001 (i.e. all of volume 22, not including the shorter communications in the Letters section). Of 206 articles surveyed, 59 included methodology that required co-iegistering of images, and thus could h~tve benefiled from the use of an automated approach. Only five of those (~8%) used some type of automated approach for locating ITPs, while 24 (~40%) appeared to use manual co-location of points (tablc l). Another 13 studies (~22%) stated that they co-registered images, but provided no detail how they did so. However, we infer from the lack of attention paid to describing geo-registration methodology that most or ,'ill of these studies did not use techniques other than the traditional manual approach. Thus, it appears likely that over 60% of the studies used manual ITP identification, while fewer than 10% of the studies used an automated approach. The remaining 30% of the studies manually registered both images to a common system, which is another situation where automated approaches could have been used. In all, more than 90% of the studies that could have used automated techniques did not. Table 1. Tallies of International Journal ~f Remote Sensing 2001 articles that co-registered images, broken down by the method used to identify ITPs. hnage co-registration method Number of papers Images separately registered to common system Manual ITP selection Manual ITP selection inferred, but not explicitly stated Automated approach Method not stated Total 17 12 12 5 13 59 Image-to-image coregistration 3469 The cause for this avoidance of automated techniques is unclear. Certainly, reviews of automated techniques have stressed the weaknesses of both categories of automated tie-point designation, Area-based methods require significant computer time (Barnea and Silverman 1972), may only deal with translational offsets (Pratt 1974), and may not be useful for data fusion between optical and non-optical sensors (Rignot et ul. 1991, Li et al. 1995). Feature-based methods require the existence of robust features that are relatively invariant between images, and also require use of sophisticated feature-extraction algorithms (Fonseca and Manjunath 1996). From the perspective of an applied user of satellite data, it may appear that the automated detection of tie-p6ints holds more risk than potential reward. We suggest that many applied users of remote sensing data could benefit from a simple automated approach. In this paper, we describe a publicly-available software package we have developed that uses a simple area-based approach for automated detection of image tic-points. We base the software on two strategies: (1) capitalize on user knowledge and pattern recognition ability, and (2) favour robustness to a variety of image conditions over optimality in any one condition. The former strategy eliminates the greatest unknowns and the costliest components of automated imagematching. The latter relies on the high speed of today's computers to overcome the time-consumption of the area-based approach. In addition to describing the software, we report on tests designed to characterize its usefulness under a range of potentially limiting conditions. 2. Methods 2.1. Automated location of tie-points Locating each tie-point is a three-step process. In the first step, an approximate tie-point pair is defined in the two images. The approximate tie-point pair is fed to the area-based correlation module that derives the precise image tie-point pair. Finally, the new pair is k, alidated. The next cycle begins by referring to theposition of the previous tie-point pair. Step 1: Locating approximate tie-point pairs. For each image pair, the user supplies the coordinates of the first approximate tie-point pair. This simple user input follows the strategy of capitalizing on human input. The match need not be a true image tie-point, only an approximation to seed the process (see section 4.7 for a brief discussion of this strategy). After the first image tie-point is found and validated (Steps 2 and 3 below), the program automatically locates the second and all subsequent approximate tie-point pairs along a regular grid pattern, working systematically left to right and up to down within the images. The spacing of the grid is adjustable. For the grids to be scaled properly, the true pixel size (pixel-centre to pixel-centre distance on the ground) of each image must be known, as well as the rotation of the images relative to each other. The user supplies both of these values. Tests of the sensitivity of the process to errors in these estimates are described in section 2.2. Again, the rationale for requiring this user input is based on the strategy of capitalizing on human strengths: the human user likely knows this basic information about the two images, and even if these two values are not already known, they can be approximated easily from simple trigonometry. This is far more efficient than designing an iterative procedure to derive these values automatically tinder any set of circumstances. 3470 R. E. Kennedy and IV. B. Cohen Step 2: Locating potential tie-point pairs. An area-based matching approach is the core of the method. A small subset (dimensions Wand H, in units of reference image pixels, adjustable by the user) is extracted from each of the two images (figure 1). The subsets are centred at the coordinates of the approximate tie-point pair supplied from Step 1. If the user has specified that the true pixel size differs between the two images, or that the images are rotated relative to one another, tile program adjusts the input image subset to match the size and orientation of the reference image subset. Once subsets have been defined, the process is identical to that described in Schowe!agerdt (1997L Conceptually, the reference image subset is anchored on a plane while the input image subset floats over it in incremental i and j pixel offset distances. At each offset, the pixel values in the intersection of the two subsets are used to calculate a spatial similarity index. As implemented in the routine, the two image subsets (hereafter R and I, for reference and image subsets, respectively) are firsl standardized to a mean of 0.0 and a standard deviation of 1.0, resulting in R" and I". For each i,j offset, the intersection regions are identified as RT.j and I]'j, and n the mean values of thc intersection regions are calculated (R~.j and I~.j). For the pixcls in the intersection, the following similarity index is calculated: